op4b: held-out memorization gate (diff-in-diff, shadow-first) - #101
Open
karpabot wants to merge 2 commits into
Open
op4b: held-out memorization gate (diff-in-diff, shadow-first)#101karpabot wants to merge 2 commits into
karpabot wants to merge 2 commits into
Conversation
- eval/memorization.py: pure diff-in-diff verdict + GPT-2-reference harness. A memorizer scores the pinned shard anomalously easier than a fresh disjoint control vs how clean public GPT-2 ranks the same two (cancels intrinsic text difficulty). Plus a tail statistic for concentrated near-verbatim recall. - Wired as op4b in judge_submission after op4; RALPH_MEMGATE off/shadow/enforce, fail-CLOSED in enforce, fail-OPEN+loud without a fresh control. Reference is cached per shard-pair so GPT-2 runs once, not per submission. - Catches a training-time held-out memorizer that every compute/arch/attestation gate AND HOSB is blind to (the fraud is in the loss surface): verified reject of the 2026-07-06 fraud king (dd -0.65), pass of the honest challenger (dd +0.28). - Fresh control MUST never overlap training; durable design draws it post-commit (recency-gating) so no frozen model can have trained on it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… was op1-local -> gate always fail-open-skipped) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
op4b — held-out memorization gate
Closes the training-time held-out-memorization class caught 2026-07-06: the fraud king (2c3d59e3) submitted a canonical checkpoint that passed every gate (compute, arch, attestation, op4) but had trained on the leaked eval shard — its best-ever val_bpb was memorized; true generalization on fresh text was worse than 124M GPT-2. Compute/arch/attestation gates and HOSB are all blind to this (the fraud is in the loss surface, not the code path).
Mechanism
Difference-in-differences vs a clean public GPT-2 reference:
DD = (meanNLL_M(P) − meanNLL_M(F)) − (meanNLL_R(P) − meanNLL_R(F))on the pinned shard P vs a fresh disjoint control F. Cancels P-is-just-easier/harder-text. A memorizer scores P anomalously easier than F against the referees verdict → DD ≪ 0. Plus a tail statistic for concentrated near-verbatim recall.Validated on real checkpoints: fraud king dd −0.65 → REJECT; honest #1593 dd +0.28 → PASS.
Wiring
op_memorization_gateruns as op4b injudge_submissionafter op4.RALPH_MEMGATE= off (default) / shadow (logsmemgate_shadow.jsonl) / enforce (fail-closed).RALPH_MEMGATE_TAU(0.15) /RALPH_MEMGATE_TAU_TAIL(0.05).Deploy
Ship shadow-first, calibrate τ, then enforce. Needs a fresh control at
eval/private/fresh_control.binthat never overlaps training — interim = fresh FineWeb-edu slice; durable = drawn post-commit (recency-gating). Validator-only (unmeasured) → plain resync, no measurement cutover.🤖 Generated with Claude Code